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What we can't do to monitor 
sustained attention "in flight" 
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Reliable ambulatory and unobtrusive physiological measures would 
enable the identification and mitigation of human performance 
decrement due to attentional lapses during the actual performance 
of safety-critical tasks such as piloting. 2 







One thing we might do to monitor 
sustained attention "in flight" 



fNIRS is an emerging optical neuroimaging technique 

• a "portable fMRI" with low spatial resolution 

• non-invasive, non-confining, ambulatory , low-cost 
and safe for long-term monitoring 

• Cognitive state is determined via pattern 
classification of functional activations 
(hemoglobin concentration changes). 



Prior Conclusions with fN IRS 


• It is possible to classify attentional state with fNIRS, but 
artifacts reduce classification prediction accuracy. 

• fNIRS is a promising candidate technique for operational 
application if obstacles to use in the field can be 
overcome. 

— susceptibility to 
motion artifact and 
physiological confound 

— need for comfortable, 
reliable head probes for 
artifact reduction and 
long-duration wear 



Harrivel, A. et al. 2011. Monitoring attentional state using functional near infrared spectroscopy: A pilot study. 

17th Annual Meeting of the Organization for Human Brain Mapping, June, 2011. Quebec City, Canada, poster #519WTh. 


Motivation for artifact remova 


improving automated real time signal 
processing to remove noise artifacts and 
improve state prediction accuracy 

other techniques in the literature rely on: 

— offline analysis 

— only features of the signal itself 

— auxiliary signals 

— known stimulus timing 

— block averaging 



Frequency Domain: the phase shift 

fNIRS is commonly performed with continuous wave optical 
instruments to calculate relative changes in [Hb]. 

In frequency domain instruments, a modulated input source 
allows for AC, DC, and phase shift measurements at each 
wavelength source channel. 

The phase shift signal is sensitive to motion: high variability 
indicates poor optical coupling due to a less stable optical 
path length. 

— If the phase is steady, it is likely the changes are due to 
activations of interest, systemic physiology or detector gain 
changes. 

— If not, absorption changes due to path length changes can 
masquerade as activations. The phase signal can indicate 
their origin for confidence in signal quality control. 
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The phase provides a direct measure of the noise 
in the optical measurement, and can be used to tune 
an online adaptive filter, such as a Kalman filter. 

LEW-18952-1: "A Novel Real Time Adaptive Filter For The Reduction Of 
Artifacts In Functional Near Infrared Spectroscopy Signals" 


Time Update (“Predict”) 


(1) Project the state ahead 

-** = A h-i + Bu k-i 

(2) Project the error covariance ahead 

P- = AP k _ lA T + 0 


Initial estimates for .V* _ ^ and _ j 


Measurement Update (“Correct”) 


(1) Compute the Kalman gain 

K t = P k H T (HP k H T + R) 


-l 


(2) Update estimate with measurement 

-** = x k + K k (z k -Hx k ) 

(3) Update die error covariance 

P k = (I- K t H)P t 


An Introduction to the Kalman 
Filter, Welch and Bishop, 2006 


Kalman Filter Parameters 

Recurrent estimates of the true signal are based on the 
measurement and on estimates of the noise. 

Application: filter more aggressively if the phase variance 
increases dramatically, indicating poor coupling and thus 
suspect [Hb] calculations. 

• R: measurement noise variance 

If large, trust the estimate more than the new measurement. 

— R=0.1 for non-noisy time 

allow large corrections to follow closely 

— R=2.0 for noisy times 

allow only small corrections, small gain K 

• Q: process noise variance 

— expected change in the process at each step 

— if larger, allows for more rapid changes in the gain, K 
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Filtered real data 




600 


Tuning the Kalman Filter Parameters 

• We set R to change with each measurement update. 

• The measurement noise is set to vary with: 

- changes in the phase variance over Is window (6 instances) 

- changes in the correlation between the phase and the [Hb] 
signal over 20s window (125 instances back) 

• A fair assumption: 

- The phase does not change in a significant way with 
physiological activation. Neuron swelling or the influx of 
new hematocytes may affect the scattering and thus the 
optical path length and phase, but not in ways this 
instrument is able to measure (time-domain spectroscopy 
requires picosecond resolution). 
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Filtered synthetic data - static 


Static R Kalman Filtering 



Q = le-4 


150 200 250 

time (sec) 


400 


11 


Filtered synthetic data - adaptive 


Phase-based Adaotive Kalman Filter 
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Filtered human subject data - adaptive 

• first application to real human subject data, 
collected during 8min of rest 

- using a frequency domain fNIRS instrument (Imagent, ISS, Inc.) 

- under IRB approval 

- motion was not induced or controlled 

- x axis now in time instances (6.25 Hz data rate) 

• here R was set to exp(phase variance) for efficient turn on/off 

• R or Q could be tuned to remove spikes independent of phase 

- spikes that do not coincide with an increase in phase may be due to 
instrument noise or perhaps exposure to ambient light, 

but if steep enough arguably are not due to the slow physiology of 
hemodynamic activations 

- optimal Q values are larger compared to synthetic traces 

- optimal parameters may vary by probe 
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Phase-based Kalman filtered human subject data:AmHbO 
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Phase-based Kalman filtered human subject data:CmHbO 



P 0 =1 , Q=0.5, S=0, instances back=20, forward=0 



Conclusions 

The phase variance provides a very good and easily-implemented 
method for removing motion artifact from fNIRS data in real time 
without an additional sensor. 

The filter works with real human subject data and can be optimized 
for automated use in the field. 

Spikes are dealt with well, but not shifts. An additional algorithm is 
needed to restore the mean after a shift. 

Optimal parameters must be determined objectively, including 
window size selection. 

— detection of longer-term slowly-varying artifacts would 
introduce a turn-on delay and be less responsive 

- small delays may be tolerable for conservative investigators who 
wish to err on the side of signal removal 
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Further work 

• This is one step of many for pre-processing fNIRS data. 

- determine the correct order of implementation 

- combine it with physiological correction 

- apply it further to human subject trial data and determine 
the effect on state classification metrics. 


• Fidelity to the measured signal when R is low and the filter is 
off should be quantified, (if Q is high this should be good) 

• Controlled probe motion with respect to the tissue or 
phantom material should be measured for comparison to the 
phase signal, (implement tiny accelerometers) 

• The adaptive tuning of Q could be explored. 
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Scholkmann, 2010 

Spline-interpolation algorithm for removing noise 
spikes in [Hb] signals detected with a moving standard 
deviation on the [Hb] signal itself. 

They simulate a slowly-varying motion artifact 
(figures 3g and 4c). 

— difficult to differentiate from hemodynamic activation 

- the phase signal can be used to differentiate these 

mechanical changes can degrade the signal via poor 
coupling, and the[Hb] signal can be relatively flat. 

— This would not trigger a moving standard deviation 
detection algorithm. 

- The phase can be monitored in an automated fashion to 
remove this signal or alert the user to reset the probe. 



MSD Amplitude MSD Amplitude 



Time [min] 



Figure 4. Results of the MARA. Parts (and show the simulated NIRI signals without MAs (gray 
lines), with MAs (dashed hlack lines) and the artifact reduced signals < hlack lines ). MSD: moving 
standard deviation s(r), T: threshold value. 
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ABSTRACT 

Functional Near Infrared Spectroscopy (fNIRS) is an emerging neurological 
sensing technique applicable to optimizing human performance in transportation 
operations, such as commercial aviation. Cognitive state can be determined via 
pattern classification of functional activations measured with fNIRS. Operational 
application calls for further development of algorithms and filters for dynamic 
artifact removal. The concept of using the frequency domain phase shift signal to 
tune a Kalman filter is introduced to improve the quality of fNIRS signals in real- 
time. Hemoglobin concentration and phase shift traces were simulated for four 
different types of motion artifact to demonstrate the filter. Unwanted signal was 
reduced by at least 43%, and the contrast of the filtered oxygenated hemoglobin 
signal was increased by more than 100% overall. This filtering method is a good 
candidate for qualifying fNIRS signals in real time without auxiliary sensors. 

Keywords: cognition, fNIRS, in-task monitoring, Kalman filter, real time, 
signal processing 

1 INTRODUCTION 

Functional Near Infrared Spectroscopy (fNIRS) is an emerging neurological 
sensing technique applicable to optimizing human performance in transportation 
operations, such as commercial aviation. FNIRS quantifies hemoglobin 
concentration ([Hb]) changes in the brain based on optical intensity measurements. 



Hemodynamic activations can be detected across the whole head. These 
ambulatory, non-invasive measurements allow in-task monitoring. Cognitive state is 
determined via pattern classification of the functional activations. Thus, we can 
watch the activations within the brain of a pilot during the safety-critical task of 
flying. Importantly, this is distinct from the use of vigilance tests which the pilot 
could undergo only while breaking from the task at hand. Such “fit-to-fly” tests may 
be passed due to short-term increases in attentional effort (Sarter, 2006) which may 
not be sustainable for long durations. During monitoring, information regarding 
cognitive state could be used to trigger appropriate risk mitigations in real time via 
changes in flight automation and information display. 

Our work continues to focus on improving techniques for applying fNIRS to in- 
task operator characterization. Optical sensing may improve upon the existing use 
of other modalities for operator characterization, especially through combination 
with them. Examples include electroencephalography and other physiological 
measures used for Augmented Cognition (Schmorrow, 2007), operator performance 
research (Schnell, 2004), and crew cognition research (Pope, 1995). 

FNIRS quantifies [Hb] changes with time based on optical intensity 
measurements of light that has scattered through the outer layers of the cortex 
beneath the optical probe. FNIRS measures the same hemodynamic changes as the 
functional Magnetic Resonance Imaging (fMRI) Blood Oxygen Level Dependent 
signal, with lower spatial but improved temporal resolution. fNIRS provides a direct 
connection to the wealth of information in the fMRI literature. We believe this 
improves the outlook for sensing more complicated states. 

2 MOTIVATION 

FNIRS works well in the laboratory, and it has been used in many 
neuroscientific research studies, including real-time classification of state using an 
extended Kalman filter and known stimulus timing (Abdelnour, 2009). However, 
implementation for monitoring outside the laboratory requires techniques that do 
not rely on known stimulus timing or block averaging. Also, no standard for the 
removal of physiological noise and motion artifact yet exists. Many opportunities 
remain for real-time applications (Zhang 2009) and for improving robustness and 
reliability. Motion artifact can be significant, and is likely to occur in operational 
environments. Field application calls for further development of algorithms and 
filters for the automation of bad channel detection and dynamic artifact removal. 
Such advances in operational use will benefit use in clinical outpatient scenarios. 

Frequency domain (FD) instruments for fNIRS measure radio-frequency- 
modulated signal intensity amplitude and offset, plus phase shift of the detected 
optical intensity signal relative to that of the source. The phase shift data provide a 
direct indication of the coupling noise associated with signal detection and thus the 
quantification of [Hb] at that time for that channel. Thus we explore the use of this 
phase measurement to drive an adaptive filter for the removal of motion artifact in 
real-time. 



Principally we are interested in avoiding human error to improve commercial 
aviation safety by identifying the loss of attentional engagement using fNIRS 
signals. The attentional monitoring system under development (Harrivel, 2010) 
could be adapted for the detection of other cognitive states by changing the location 
of the optodes and adjusting the classifier training data and parameters. Our 
previous work has demonstrated improvements in the comfort of optical head 
probes to enable monitoring for over an hour (Harrivel, 2009), and has shown on 
average 70% accuracy for real time classification of attentional state using Support 
Vector Machines with training data free of artifact (Harrivel, 2011). To improve 
hemodynamic activation measurements and classification accuracy, we continue to 
implement existing cutting edge techniques while exploring novel headgear and 
data processing methods to provide such clean data in real time during operational 
use in the field. Here we introduce a filter based on the FD phase measurement for 
real-time signal quality detection and improvement. This new filtering technique is 
well- suited to improving the quality of fNIRS signals in any application where 
motion is an inherent part of the task to be performed during monitoring. 

3 RATIONALE 

3.1 Optical rationale 

Optical intensity is attenuated by absorption and scattering along the optical path 
length (the light’s path through the head). The phase is sensitive to motion because 
of its dependence on the source-detector separation and the index of refraction of 
the tissue through which the light travels. Both affect the optical path length. An 
increase in the actual optical path length causes greater absorption. This can lead to 
overestimation of the change in [Hb] if the path length is assumed to be constant. 
High variability of the FD phase shift can indicate inconsistent coupling at the 
optode-scalp interface, which changes the separation distance and possibly exposes 
the detector to ambient light (which has no consistent phase shift with respect to the 
source). Shifts in the phase also can indicate motion of a probe along the scalp 
surface. As the probe senses different volumes of tissue, changes in intensity are 
caused by changes in the optical scattering and absorptive properties of the tissue. 
This also contributes to probe relocation errors and inter-subject variability. 

If the phase and [Hb] changes are correlated, it is likely the [Hb] changes are 
due to motion-induced changes in path length or absorption. This is because the 
increased phase indicates a larger path length, which causes more absorption and 
lowers the detected intensity, which finally increases the calculated [Hb]. If the 
phase is steady, it is likely the calculated [Hb] changes are due to activations of 
interest, systemic physiology or detector gain changes. Thus, the phase can be 
monitored to automate signal removal or alert the user to reset the probe. 

This filter assumes the phase does not change significantly with physiological 
activation. Neuron swelling or the influx of new hematocytes may affect the 
scattering and thus the optical path length and phase, but not on time scales 



accessible to FD instrumentation. 

This filtering method is most appropriate for the Frequency Domain Multiple 
Distance (FDMD) method where FD instrumentation is used to calculate optical 
absorption and scattering properties so absolute [Hb] values can be determined 
(Fantini, 1999). In this case, the filter would be applied to the measured optical 
intensity, and anti-correlation between the phase and the intensity could be used. 
This can be explored in future work. Alternately, in cases where relative measures 
suffice while probe footprints on the head must be small and the signals must be 
reliable, it can be used to filter the relative [Hb] changes calculated using the 
Modified Beer Lambert Law. The phase signal used should be for the wavelength 
that matches the hemoglobin species of the trace being filtered. This is the case 
presented here. 


3.2 The Kalman filter and its suitability 


Kalman filtering (Kalman, 1960) is a technique for estimating the state of a 
linear discrete dynamical system. The Kalman filtering implementation used for this 
work assumes that the unknown true value x k and measurement value z k of a system 
at time k is determined by the model 

x k = "b w Sc - 1 


s k — "b 

where A and H are known linear operators. A and H were each assumed to be an 
identity operator for this work. In this way, we are assuming that the process under 
study is simply the contamination of the data by additive noise. The variables w k 
and v k are process and measurement noise, respectively. The distributions of these 
two noise sources are assumed to be 

wfe Q) and A r (CL R), 

where N(ju, o 2 ) is a normal probability distribution with mean ji and variance a 2 . The 
Kalman filter produces estimates, by means of the explicit recurrence relations, of 
the data 
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and the variance of the filter 
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beginning with the initial condition x" 0 = z 0 and p 0 = 1. Because of the explicit 
nature of these computations, no iterative solver or optimization procedures are 
required for filter implementation. Furthermore, the computed estimates x\ are 
optimal in the mean squared error sense. More specifically, 

x k = argirdu E[lx k -x k \ 2 ) 

for each time where E is the mathematical expectation operator. The optimal 
properties of the Kalman filter estimates along with the simplicity of its 
implementation (Zarchan, 2005) make it an excellent choice for the smoothing of 
fNIRS data traces. 


Values for the variances of the process and measurement noise ( Q and R) must 
be determined before the Kalman filter can be implemented; since they are required 
in the calculation of each recurrence relation (each k). These values can be 
interpreted as tuning parameters for the Kalman filter. A larger R weights the data 
estimate and distrusts the measurement to produce a smoother output, while a 
smaller R follows the measured signal more closely. Q can be increased to allow 
more dynamic changes in the estimate of the system. 

We propose that the phase channel § associated with a [Hb] channel x" is a 
natural choice of information for computing an appropriate value for R at each time. 

4 METHODS 

Synthetic data were created to mimic [Hb] and phase shift traces with a data rate 
of 6.25Hz for 400s to demonstrate the feasibility of this filter. Four [Hb] time series 
were simulated: relative changes in both oxygenated and deoxygenated (reduced) 
hemoglobin species concentration ([HbO] and [HbR]) for two source-detector 
separations, which we label close and far. A close source can be used to measure 
systemic physiological contributions to the signal of interest (biological “noise”) on 
a per-sensor basis (Zhang, 2009) by interrogating shallow tissue (not cortex). The 
far source interrogates both superficial and cortical tissue. 

The input trace to be filtered was created by adding non-dynamic physiological 
signals for cardiac at 1Hz, respiration at 0.25 Hz, and Meyer waves at 0.1Hz to the 
close source traces. Functional activations, and smaller physiological contributions, 
were added to the far source traces. Random noise was added to all traces. In figure 
3, the input [Hb] trace is shown (top, black) after within-species subtraction of the 
close source signal from the far source signal, then subtraction of [HbR] from 
[HbO]. Functional activations were simulated with gamma-variant functions as in 
Abdelnour, et al. (2009) eq. 3, with the [HbR] amplitude being -1/4 of [HbO]. 

Four different types of motion artifact were simulated. These are outlined in 
Table 1. No motion artifact is simulated from 0s to 50s. Artifact from probe slip 
(across skin with no air gap) was simulated between 50s and 100s by allowing the 
phase to vary randomly over tens of degrees. Artifact from decoupling (an air gap at 
the detector interface) was simulated between 100s and 200s with phase varying 
over hundreds of degrees. Artifact from a probe bump was simulated with a spike in 
the phase at 225s, while artifact from a bump and relocation was simulated with a 
spike and a step increase of 10 degrees at 325s. Phase changes on these orders of 
magnitude do occur in real data. [Hb] signal which covaried with the phase was 
generated for the slip and the relocation bump artifacts. 

In the case where a noisy signal could appear to be a [Hb] change (50s - 200s), 
filter performance is quantified as percentage of signal mean reduction. In the cases 
where motion noise obscures the activation of interest, performance is quantified as 
contrast-to-noise ratio (CNR) improvement between the unfiltered and the filtered 
signal. CNR is defined here as mean activation signal minus mean reference signal 
(0s to 50s), divided by the square root of the sum of the variances in those signals. 



4.1 Mathematical Implementation of the Kalman filter 


To implement the Kalman filter, the measurement noise variance, R, can be set 
to a function, possibly non-linear, of the variance in the phase over a window of 
time prior to that instance. The use of the phase differentiates our method from 
other implementations of Kalman filtering for motion artifact removal (Izzetoglu, 
2010). Here, the tuning parameter R was set to the variance of the phase in a rolling 
window of Is prior to the current instance, plus the correlation of the phase with the 
[Hb] trace being filtered during 20s prior (which turns on at 20s). The contributions 
of the variance and correlation were linearly scaled. The absolute value of the 
correlation was added if it was greater than a threshold of 0.3. Thus only high 
correlations impact R. This threshold has not been optimized, and can be raised for 
less aggressive filtering. 

4.2 Parameter Selection 

Parameters were selected empirically for this pilot study to maximize signal 
retention and artifact rejection. Different scale factors and rolling time window sizes 
were explored for both the variance and the correlation. The scale factors were 
selected to bring the magnitude of R into the useful range of 0 to 5. R should be 
small (~0) to allow desired changes in signal to be retained, and large to remove 
artifact. R on the order of 2 or more was found to remove artifact effectively (as in 
figure 3, 100s - 150 s). 

Increasing window size increases signal plateau due to filter turn off delay, as 
seen in figure 3 at 340s. It is due to correlation contributing to R even after it has 
passed in time. This may be tolerable for conservative investigators who wish to err 
on the side of signal removal. Shorter windows may be less able to detect artifact. 
Figures 1 and 2 show improved rejection (signal mean reduction) of the probe slip 
(50s - 100s) and the probe relocation (325s) artifacts attained by allowing the 
correlation to contribute to R. 

Q was set to le-4 or le-5 as noted in the figures. If larger, high frequency spikes 
are not removed. If smaller, desired signal changes are attenuated. The initial state 
estimate was set to z 0 = 0, and the initial variance of the filter was set to p 0 = 1 . All 
of these parameters are available for adjustment and optimization, with the objective 
of maximizing the rejection of artifact and the retention of good signal. 

In real application, optimal parameters may depend on source-detector 
separation. For example, the signal to noise ratio of the phase itself depends on the 
source-detector separation (Dehaes, 2011). Ideally, the [HbO] close and far, and 
[HbR] close and far traces should be filtered with their own parameters. 
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Figure 1 Calculated R (bottom) used to filter the [HbO] trace (top, filter output in negative). The 
phase variance from a rolling window looking back Is is used to calculate R. Q was set to 1e-4. 
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Figure 2 Calculated R (bottom) used to filter the [HbO] trace (top, filter output in negative). 
Both the phase variance from a rolling window looking back Is, and the correlation from a rolling 
window looking back 20s, are used to calculate R. Q was set to 1e-4. 


5 RESULTS 

After filtering, the mean of the unwanted peak between 50s and 100s is reduced 
by 43%. That between 100s and 200s is completely removed. The reference signal 
mean is reduced by 9%. CNR is increased by more than 100% overall. The results 
are listed in Table 1. The filter output trace is shown in figure 3 (negative, top). 
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Figure 3 Calculated R (bottom) used to filter the [Hb] trace (top, filter output in negative). Both the 
phase variance and the correlation are used to calculate R. Q was set to 1e-5. 

Table 1 Phase-based adaptive Kalman filter performance by motion type 


Time 

Motion 

simulated 

Phase shift signal 
feature 

Filter Performance Result 

Os- 

50s 

none 

(reference) 

none 

(reference) 

• CNR increase: 125% 

• signal mean reduction: 9% 

50s- 

100s 

probe 

slip 

slow changes 
correlated with [Hb] 

• unwanted signal mean 
reduction: 43% 

100s- 

200s 

probe 

decoupling 

very high variance 

• unwanted signal mean 
reduction: 100% 

200s- 

300s 

transient 
probe bump 

spike at 225s 

• CNR increase: 144% 

• signal mean reduction: 12% 

300s- 

400s 

probe bump, 
relocation 

spike and step 
increase at 325s, 
correlated with [Hb] 

• CNR increase: 204% 

• signal mean increased 1% 

• late filter turn off extends 
activation by 10s 


6 DISCUSSION 

An advantage of this filtering method is that it only comes on when needed 
according to actual conditions without adding a channel or auxiliary sensor. The 
phase provides an objective indication of whether [Hb] changes are due to motion 
without relying only on the [Hb] trace. Also, the phase can simply be used to 









indicate data quality for complete removal during post-processing. 

A disadvantage is the introduction of time delays. The filter can only change the 
estimate so quickly, and delays can be on the order of seconds when the filter is on 
(high R). Filter turnoff delay is also a problem, as discussed in section 4.2 and as 
evidenced by the artificial increase in signal mean between 300s and 400s. Q may 
be increased and time windows shortened to reduce delays. 

Other methods for real time artifact removal include simply smoothing, using 
other signal correlations, or using a moving standard deviation. With smoothing, 
unwanted signal is not rejected, and the temporal resolution advantage of fNIRS is 
eroded in all instances. The Kalman filter can be adjusted to quickly follow non- 
noisy data by instance. However, smoothing does not introduce delay. 

Cui, et al. (2010) present a successful method for removing motion artifact from 
fNIRS signals which takes advantage of the anti-correlation between [HbO] and 
[HbR] inherent in functional activation. However, the phase-based filter removes 
motion artifact from both channels intended to measure functional activations and 
channels for systemic physiology, which lack neural activation. 

A moving standard deviation measurement on the [Hb] signal itself has been 
used to detect spikes for removal (Scholkman, 2010). However, mechanical changes 
can degrade the signal via poor coupling resulting in a relatively flat [Hb] signal. 
This would not trigger a moving standard deviation detection algorithm. 


7 CONCLUSIONS AND FUTURE WORK 

The phase is a good candidate for qualifying fNIRS signals in real time, 
enabling automated bad-channel detection and motion artifact removal without 
auxiliary sensors. Successful future work would increase the amount of motion 
which can be tolerated while obtaining useful fNIRS signals. This real-time filtering 
technique relies upon frequency- domain instrumentation, but is well- suited to 
improving the quality of systemic physiology and functional activation 
measurements with fNIRS in any operational environment where motion is an 
inherent part of the task being performed. 

Further parametric study is needed to minimize delays and maximize the 
performance of the filter for data collected with human subjects. The investigation 
of non-linear functions of the phase variance and the correlation between the phase 
and the [Hb] is planned. The filter should be tried on the measured optical intensity 
with the FDMD method. Human subject studies to quantify the motion of the probe 
with respect to the head during the phase shift measurement should be undertaken. 
Ultimately, cognitive state classification could be performed with this filter to 
determine the effect on classification accuracy. 
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